The paper titled "BitQ: Tailoring Block Floating Point Precision for Improved DNN Efficiency on Resource-Constrained Devices" addresses the challenges associated with deploying deep neural networks (DNNs) on devices with limited computational resources. DNNs are widely recognized for their effectiveness in various cognitive tasks, including image classification, object detection, and scene segmentation. However, their high computational complexity and substantial memory requirements often hinder their real-time application on embedded platforms. To mitigate these issues, the authors explore block floating point (BFP) quantization, a compression technique that reduces the memory and computational demands of DNNs. BFP quantization is particularly advantageous because it can effectively capture the diverse data distributions inherent in DNN models. Despite its benefits, previous research in this area has typically relied on empirical methods to determine block sizes and precision levels that maintain accuracy, which may not be optimal. In response to this gap, the authors propose a novel analytical modeling framework called "BitQ." This framework is designed to optimize the implementation of BFP for DNN inference on resource-constrained devices. The authors formulate an optimization problem that seeks to identify the ideal block size and bitwidth distribution, balancing the trade-offs between accuracy and performance loss. The experimental results presented in the paper demonstrate that DNNs utilizing the optimized bitwidth allocation provided by BitQ outperform those using a uniform bitwidth setting. This optimization leads to more efficient computation while preserving accuracy across well-known benchmarks. The authors have made their source code and data publicly available, facilitating further research and application in this domain.

1 Bit language models are exciting. This work shows how to quantize the Linear layers of a language model without sacrificing performance. This can result in a 70B model running on consumer GPUs.

DecoupleQ is a quantization approach that significantly enhances the accuracy of large models at ultra-low bit levels. This method restructures the quantization process by splitting model parameters into integer and floating-point parts that are then optimized using traditional methods.

Large language models are demanding more and more energy and computational power as they get better. These models need to shrink to become cheap, fast, and environmentally friendly. Researchers use a process called quantization to compress networks by reducing the precision of their parameters. They are now pushing the envelope to single bit, producing models that are faster and more energy efficient than their full-precision counterparts. The quantized versions of the models perform almost as well as their original versions.

Microsoft has released a set of GPU accelerated kernels for training BitNet style models. These models have substantially lower memory cost without much drop in accuracy.

VPTQ, or Vector Post-Training Quantization, is an innovative algorithm developed by Microsoft aimed at achieving extreme low-bit quantization for large language models (LLMs). This method allows for the compression of models, such as the 70 billion and even 405 billion parameter models, to a mere 1-2 bits without the need for retraining, while still maintaining high accuracy. The algorithm is designed to be lightweight, taking approximately 17 hours to quantize a 405 billion parameter model like Llama-3.1, and it offers agile inference capabilities with low decoding overhead, ensuring optimal throughput. The challenge of scaling model sizes has led to increased interest in low-bit quantization techniques, particularly due to the redundancy found in LLM weights. Traditional scalar-based quantization methods struggle to achieve effective low-bit representation due to numerical limitations. In contrast, VPTQ utilizes vector quantization, which compresses weight vectors into indices through lookup tables, enabling significantly lower bit-width quantization while preserving model performance. Early results from the VPTQ tech report indicate that the algorithm outperforms existing methods in terms of accuracy and throughput across various model sizes. For instance, the quantization results for LLaMA-2 models show improved performance metrics, including lower memory usage and faster token processing rates, demonstrating the effectiveness of VPTQ in practical applications. To implement VPTQ, users need to ensure they have the appropriate dependencies, including Python 3.10 or higher, and specific versions of libraries such as PyTorch and Transformers. The installation process involves setting up the CUDA environment and executing a pip command to install the VPTQ package. The repository also provides examples for generating text using pre-trained models, launching chatbots, and utilizing the Python API for model interaction. However, it is important to note that the repository serves primarily as a method for model quantization, and the performance of models provided by the open-source community cannot be guaranteed. Future plans for VPTQ include merging the quantization algorithm into public repositories, submitting the method to various inference frameworks, and enhancing the implementation of the inference kernel. The project is led by a team of contributors who acknowledge the foundational research that inspired their work. While VPTQ shows promise, it is intended for research and experimental purposes, with limitations regarding its application across different languages and tasks. The project encourages contributions and adheres to a code of conduct, ensuring a collaborative and respectful environment for developers and researchers interested in advancing the field of model quantization.

The powerful DeepSpeed training library from Microsoft has an update that allows models to use 6 bits per parameter. This can speed up inference well over 2x.

Nvidia's new Blackwell chip demonstrated top per GPU performance in MLPerf's LLM Q&A benchmark, showcasing significant advancements with its 4-bit floating-point precision. However, competitors like Untether AI and AMD also showed promising results, particularly in energy efficiency. Untether AI's speedAI240 chip, for instance, excelled in the edge-closed category, highlighting diverse strengths across new AI inference hardware.

The team at Snap Research was able to reduce the size of the Stable Diffusion UNet model from 1.72 GB down to 219MB while increasing performance with their new quantization scheme. The quantization method is somewhat complex, but paints a strong path forward in running generative models on consumer hardware.

The GitHub repository titled "PerCo" by Nikolai10 presents a PyTorch implementation of a novel image compression technique aimed at achieving perfect realism at ultra-low bitrates. This work is based on the paper "Towards Image Compression with Perfect Realism at Ultra-Low Bitrates," which is set to be presented at ICLR 2024. The repository distinguishes itself by utilizing Stable Diffusion v2.1 as its latent diffusion model, contrasting with the original work that relied on a proprietary pre-trained model. The project is actively under development, with several updates already made. Notable improvements include fine-tuning the entire U-Net architecture, which has led to enhanced results, and the release of pre-trained models. The repository also documents various experiments, including ablation studies that explored different techniques without achieving significant improvements. Visual comparisons of the compression results on the Kodak dataset illustrate the model's performance at the lowest bit-rate, showcasing reconstructions that reflect uncertainty about the original images. The repository provides quantitative performance metrics, indicating that while the PerCo (SD v2.1) model achieves competitive perceptual results, it sacrifices some image fidelity compared to the official model due to fewer training steps. Installation instructions are provided, along with guidance for training, inference, and evaluation. The project uses the OpenImagesV6 dataset for training and offers a simplified Google Colab demo for ease of use. Future plans include enhancing compression functionality, integrating additional datasets, and refining the training pipeline. The file structure of the repository is organized into directories for Docker functionality, Jupyter notebooks, evaluation data, and source code. The project acknowledges various libraries and frameworks that inspired its development, including HuggingFace's Diffusers and Transformers, as well as other tools for data compression and neural network research. Overall, the PerCo repository represents a significant step forward in the field of image compression, aiming to balance the trade-offs between perceptual quality and image fidelity at extremely low bitrates. The project is licensed under the Apache License 2.0, encouraging collaboration and further development within the open-source community.

Answer AI has released a new FSDP/QLoRA training tool that makes it possible to train 70B parameter models on consumer GPUs. It has open sourced the code and made it easy to run locally or on runpod.

Meta's LLaMA3, a leading large language model, is being tested for its efficiency in low-bit scenarios, often essential in systems with limited resources. This study, available on GitHub and Hugging Face, aims to refine and improve quantization strategies for future large language models.

Google designed the TPU v1 for fast, cost-effective inference using trained neural network models at scale. Its key feature is a focus on tensor operations, specifically matrix multiplications, which are core to neural network computations. The TPU v1 is 15-30x faster than contemporary CPUs/GPUs for inference. It has 25-29x better performance per watt than GPUs.

LLMs demand a lot of energy, but researchers are finding ways to reduce their size by quantization - representing model parameters with only 1 or -1. The two main approaches are post-training quantization (PTQ) and quantization-aware training (QAT). PTQ is currently more popular. Despite lower perplexity scores, 1-bit LLMs are much more energy efficient and faster on customized chips.

An impressive array of open models that approach the frontier of performance. Specifically, they have strong performance on code, math, structured output, and reasoning. The Qwen team has also released a suite of sizes for a variety of use cases.

The article discusses the development and performance of a set of tiny test models trained on the ImageNet-1k dataset, created by Ross Wightman and published on Hugging Face. These models represent various popular architecture families and are designed for quick verification of model functionality, allowing users to download pretrained weights and run inference efficiently, even on less powerful hardware. The models are characterized by their smaller size, lower default resolution, and reduced complexity, typically featuring only one block per stage and narrow widths. They were trained using a recent recipe adapted from MobileNet-v4, which is effective for maximizing accuracy in smaller models. While the top-1 accuracy scores of these models may not be particularly impressive, they are noted for their potential effectiveness in fine-tuning for smaller datasets and applications that require reduced computational resources, such as embedded systems or reinforcement learning tasks. The article provides a detailed summary of the models' performance metrics, including top-1 and top-5 accuracy scores, parameter counts, and throughput rates at a resolution of 160x160 pixels. The results indicate that the models, while small, can still achieve reasonable accuracy levels, with some models performing better at a slightly higher resolution of 192x192 pixels. Additionally, the article outlines the throughput performance of the models when compiled with PyTorch 2.4.1 on an RTX4090 GPU, showcasing the number of inference and training samples processed per second under different compilation modes. This data highlights the efficiency of the models in terms of speed, which is crucial for real-time applications. The article also delves into the unique architectural variations of the models, providing insights into their design and the specific components used in each. For instance, the ByobNet combines elements from EfficientNet, ResNet, and DarkNet, while the ConvNeXt models utilize depth-wise convolutions and different activation functions. The EfficientNet models are noted for their use of various normalization techniques, including BatchNorm, GroupNorm, and LayerNorm. Overall, the article invites the community to explore potential applications for these tiny test models beyond mere testing, emphasizing their versatility and the innovative approaches taken in their design.

MobileLLM optimizes sub-billion parameter language models for on-device use cases.

The paper titled "MaskBit: Embedding-free Image Generation via Bit Tokens" presents advancements in the field of image generation, particularly focusing on class-conditional image synthesis. The authors, Mark Weber and his colleagues, explore the potential of masked transformer models as a viable alternative to traditional diffusion models. Their approach is structured around two main contributions. Firstly, the authors conduct a thorough examination of Vector Quantized Generative Adversarial Networks (VQGANs), leading to the development of a modernized version of this model. This updated VQGAN is designed to enhance transparency and reproducibility in image generation, while also achieving performance levels that are competitive with the current state-of-the-art methods. The authors emphasize the importance of making their findings accessible, revealing previously undisclosed details that could benefit future research. Secondly, the paper introduces a novel generation network that operates directly on bit tokens, which are binary quantized representations of data. This embedding-free approach allows for efficient image generation while maintaining rich semantic information. The results demonstrate that this method achieves a remarkable Fréchet Inception Distance (FID) score of 1.52 on the ImageNet 256x256 benchmark, indicating a high quality of generated images. Notably, the generator model is compact, consisting of only 305 million parameters, which contributes to its efficiency. Overall, the study highlights significant advancements in image generation techniques, showcasing the effectiveness of embedding-free methods and the potential of bit tokens in producing high-quality images.

One drawback of modern transformers is that each token uses the same amount of predictive compute. However, some tokens are much easier to predict than others. This work from DeepMind allows models to exit early during generation to spend less flops on certain tokens, effectively opening the door to dynamic compute - with a fixed maximum. The results are 50% fewer flops at generation time for equivalent performance.

Researchers claim to have developed a method of running AI models more efficiently that involves eliminating matrix multiplication from the process. A fundamental redesign of the neural network operations that are currently accelerated by GPU chips, the method could have deep implications for the environmental impact and operational costs of AI systems. It challenges the prevailing paradigm that matrix multiplication operations are indispensable for building high-performing language models. The approach may outperform traditional large language models at very large scales, but this has not been tested due to computational constraints.

This article covers a cross-browser local LLM inference engine that uses a new quantization technique and WebAssembly to deliver fast LLM inference.

Nvidia has released a dataset and recipe along with a high quality paper about training reward models to align model output to human preferences.